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Authors ? Abstract 



The purpose of the present study was to explore growth in writ- 
ten composition in tho community college by using a group-devised 
scoring key to score pre- and post-compositions. The study was con- 
ducted in three community colleges in Southern California, with each 
student r s writing ability being measured by pre- and post-composi- 
tions written during the first and last weeks of an eighteen-week 
semester , 

The pro- and post-compositions were collected , given blind code 
numbers, and randomly selected to form packets of approximately 
twenty-five compositions. These packets were then distributed for 
scoring to the twenty-one English instructors in the three colleges. 
The final sample included 252 pairs of pre- and post-compositions. 

Using the scoring key they had developed, the instructors 
graded their packets of compositions. Group means were computed on 
a pre- and post-basis for the total sample, as well as for each of 
the three colleges, on the major section of the scoi*ing key. A 
cross-tabulation of pre- and post-scores for each of the fifteen 
items in the scoring key was conducted to analyze changes in writing 
ability. 

The inter- rater reliability of the instructors w r as checked 
during the development of the scoring key and again during the final 
scoring , 

No significant changes in writing ability were detected in this 
study through a comparison of pre- and post-means for the total sam- 
ple, or for any of the three colleges, as indicated by a t-test for 
correlated samples. An analysis of the individual score changes 
indicated that almost all student scores changed slightly during 
the semester. 

This study supports the use of a cooperatively developed 
scoring key to reduce rater bias. It does not support the assump- 
tion that community college students improve their writing skills 
following eighteen weeks of instruction in composition. 
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Chapter 1 
INTRODUCTION 

Does anyone learn to write in college? How would one go about 
seeking pertinent data? Ask the instructors? Search the dean's 
files? Poll the students or check their grade point averages? 

All available answers suffer from the limitations of bias, dis- 
torted perception, and, above all, inadequate information. Grade 
marks earned in freshman composition courses, for example, say 
little about writing ability. The classes may be based on tradition- 
al grammar, structural linguistics, literature, rhetoric, logic, 
semantics, communication in the mass media, public speaking, or any 
combination thereof (Kitzhaber, 1963), In some cases, instruction 
and practice in writing are not included in the course at all and 
marks assigned are related instead to verbal performance, responses 
to quick score exams, or facility in classroom discussion. 

Other sources shed no light whatsoever on the question of stu- 
dent learning. Deans' files typically include data on numbers of 
students who transfer to other institutions and the grades they 
earn. When students and instructors are queried, their answers are 
usually, "I feel I learned to write" and "I think our students are 
writing better now than when they enrolled." The extent to which 
the questioner is satisfied with those responses depends on his 
faith in the accuracy of student memory and instructor perception. 

If there are answers to these questions, they must be based on 
certain premises. Among the premises accepted by this study is that 
colleges are supposed to cause or allow learning to occur. For pur- 
poses of this study, learning is defined as changed capability for 
or tendency toward acting in particular ways. It is assumed that, 
when students write better compositions at the end of the course 
than they did at the beginning, they have learned. It is also as- 
sumed that the instructor is the person who should assess the level 
of learning attained by his students. To do this he must define 
what he will accept as evidence of learning, but individual instruc- 
tor assessment is not adequate for the measurement of student learn- 
ing as defined by this study. For a variety of reasons, it is im- 
portant that the instructors of several classes agree on the way 
learning shall be measured. 

It is not difficult to defend the validity of these assumptions. 
The definition of learning as "changed capability" is widely accepted 
(Hilgard and Bower), The pre-post design for assessing learning is 
also well known (Campbell and Stanley), and the necessity for in- 
volving the instructors in determining student learning has been fre- 
quently iterated (Seri veil) . A prime reason for this involvement is 
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that instruction-wide testing procedures do not often provide infor- 
mation useful in revising the English courses themselves. Many in- 
stitutions administer entrance examinations and use the results for 
student placement in various curriculum levels. The student com- 
pletes a "verbal ability" test or submits a writing sample and is 
then placed in a course ostensibly geared to his level of proficien- 
cy. Any follow-up that may be conducted correlates entrance test 
scores with the grade marks earned in the courses themselves. This 
practice, however, has little if any relationship to a student’s 
having learned to write, and fails to yield data useful to the in- 
structors themselves. 



Within the schools, composition scoring is typically conducted 
by individual instructors in the confines of their own courses, but 
the practice suffers from bias--intentional or otherwise — and fre- 
quent distortion. A single instructor's assessment of his students' 
compositions yields a measure that may be of value to him but that 
cannot be compared with scores of students in other courses . Each 
instructor applies his own criteria for his own purposes. 

The array of composition scoring devices and procedures current- 
ly available makes it difficult to assess student writing ability for 
purposes of instructional improvement. Many group composition scoring 
procedures have clear directions and high inter-rater reliability yet 
yield only global measures (Lambert, 1969) . A general rating of the 
relative worth of a composition may be useful in deciding college 
admission or curricular placement, but it provides little information 
on which an instructor may base changes in instructional procedures 
or emphases. 

The problem of determining whether or not (and the extent to 
which) students learn to write as a result of attending the junior 
college in particular is far from being resolved. During the past 
three years, few research documents received and processed at the 
ERIC Clearinghouse for Junior Colleges address themselves to that 
issue. The dearth of such studies is noted in several publications 
resulting from conferences sponsored by the National Council of 
Teachers of English (Archer, 1965; Braddock, 1963; Weingarten and 
Krueger, 1965) , These books call for more research on composition 
scoring, instructional procedures, and variables pertaining to stu- 
dents' learning to write in the junior college. Current pedagogy 
demands validated procedures for assessing their results (Zoellner, 
1969) . Research models including designs that can be used by in- 
structors are necessary for these studies. 

This design stems from certain definitions of and philosophical 
positions on education. The over-riding position is that education 
is a process of moving people from one level of capability or tenden- 
cy to another. Thus, education is not seen as providing an environ- 
ment in which something of unknown effect may or may not occur; 



rather, education is the bringing about of change. Within the com- 
munity college, it is the instructors who should predict and define 
the nature of that change and assess the effects of their instruc- 
tional process. 

A feasible design was prepared and employed by the principal 
investigator of this project in a study conducted during the 1968-69 
academic year. This design was modified for use in this study. 

The general hypotheses tested by this research are: (1) feasi- 

ble procedures for scoring English compositions can be developed* 

(2) significant differences in writing improvement can be measured 
and differentiated. The data were collected from sample populations 
in three colleges: (a) suburban, with 6463 students; (b) urban, with 

5432 students; (c) rural, with 1335 students. 

Two similar topics were selected for the pre- and post-test. 
Instructions were simple and uniform. During the first and last- 
week of classes, students received a blue book with instructions to 
write on the indicated topic. Their compositions were collected and 
distributed randomly to instructors for rating. Each composition 
was scored blind and coded on a separate sheet. Pre- and pest- 
compositions were mixed together before being distributed. Scoring 
sheets provided the basis for analysis. A second reader scored a 
random 100 compositions from the total sample. An internal consist- 
ency of the 16 scores was tested by item analysis. A discrimination 
co-efficent of ,30 was considered satisfactory and .50 was very 
good . 
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Chapter 2 
PREVIOUS RESEARCH 



Overview 

It is perhaps misleading to speak of composition research, since 
the English profession has accumulated far more unanswered questions than 
it has empirically collected answers. Noting this , Kit zhaber (March 
1962, p* 444] has suggested that the English teaching profession urgently 
needs 15 to 20 years of careful research on the problems of composition. 

Such a statement is not meant to imply that researchers have completely 
avoided the problems of composition, but it does indicate the magnitude 
of the task ahead, 

A measure of research activity since 1902 can be obtained by scanning 
the 504 composition studies cited by Braddock, Lloyd -Jones , and Schoer in 
their thorough survey. Research in Written Composition . Unfortunately, 
their review of research uncovered little of value ; and the reviewers ex- 
pressed their disenchantment with the state of the art in this forthright 
manner : 

Today* s research in composition, taken as a whole, 
may be compared to chemical research as it emerged 
from the period of alchemy: some terms are being 

defined usefully, a number of procedures are being 
refined, but the field as a whole is laced with 
dreams, prejudices, and makeshift operations (1963, 
p.S). 

It is significant that there has been no shortage of pleas for re- 
search from within the English profession (Archer, 1965) . The National 
Council of Teachers of English issued several recent calls for further 
research on the fundamental questions associated with written composition* 
Reviews such as Research and the Deve lopment of English Programs in The 
Junior College (Archer and Ferrell^ - 1965), with its excellent survey of 
the general problems of teaching English, and English in the Two-Year 
College (Weingarten and Kroeger, 1965), which spotlights more specifically 
the difficulties in teaching composition, must certainly be given credit 
for their searching honesty. 

Measuring Change in Writing Ability 

Attempts to measure change in writing ability have been generally 
disappointing; however, in the few studies relevant to this investigation, 
one or more causes for failure can be identified. Eurich ! s study in 1932 
is a case in point. Using the Van Wagenen English Composition Scale to 
score 54 freshmen on pre-and post-test themes, he attempted to evaluate 
the effectiveness of three months of English instruction at the University 
of Minnesota. His results show that 35 students made no gains, or declined 
slightly, while 19 made only slight gains. Enrich 1 s conclusion was succinct 
"There is no evidence students improve their ability to write in composition 
(March 1932, p. 215). 
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Two points in Eurich's study deserve further comment. First, he 
notes in his summary of the evidence regarding composition effective- 
ness that : 



The limited evidence available seems to indicate 
that improvement may be found primarily on the 
materials of drill or practice. In other words, 
general improvement does not necessarily follow 
specific training. To be effective, training 
in composition should be directed toward definite 
and specific ends (Ibid, p. 215). 

If the scoring key used in Eurich's study had been employed throughout 
the period of the experiment, the scoring key might have served to focus 
student and instructor attention on ’’definite and specific ends;" with 
this change in procedure, gains in writing skill might have been identi- 
fied . 



The disappointing results obtained in Eurich's study could also be 
caused by the short duration of his experiment. It may well be that more 
than three months of composition instruction is required to cultivate 
significant changes in writing skill (Scannel and Haugh, June 1968, p.4). 
Miller (1958) did observe some growth in the writing ability of college 
freshmen over a period of one year. However, he reports that the improve- 
ment after a year of English was no greater than from D+ to C-, and that 
the majority of his 200 students received the same rating on the first and 
final papers. It should be noted, however, that the low reliability of the 
scoring key used to grade pre- and post -themes was a contributing factor 
to Miller’s results. 

Related to the findings of Eurich and Miller are the observations of 
Fellers (1953), who analyzed the type and frequency of composition errors 
made by 80 students during their last semester in high school against 
those made by the same students after one semester of college. Fellers 
found little change in type or frequency of composti on errors , even though 
all students had taken a college composition course. Thus, while these 
studies by Eurich, Scannel and Haugh, Miller, and Fellers seem to attest 
to the persistence of student errors in composition and to the apparent 
difficulty in causing changes in student writing skills, their conclusions 
may not be valid when instruction focuses on definite composition skills. 

Braddock and Statler (1968) tested the effects of an English writing 
course versus the effects of no course with 79 matched pairs of freshmen 
at the University of Iowa over a two-year period. In this study, scoring 
was done by two groups of raters, each using a different scoring scale; 
the results showed no significant differences for either group of students. 
However, these results are equivocal, since neither rating scale produced 
high rater reliability, and particularly since the raters were able to 
identify pre- and post-themes. An evaluation procedure must lessen the 
effects of both these problems. 




Other studies of change in writing ability have even reported de- 
clines. Sutton and Allen (1964) studied the effect of practice and 
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evaluation on improvement in written composition. Using control and ex- 
perimental groups of college freshmen, they elicited twelve themes from 
each group. Five readers read each theme twice to rate and rank it in 
relation .0 the other eleven themes by each writer. Rankings were given 
on five criteria (ideas, mechanics, wording, form, and flavor) and a five- 
point scale was used. Sutton and Allen found declines for each group and 
for the groups combined. The frequency of writing themes that were neither 
commented on nor returned to the students may have created an attitude of 
boredom and impatience among the students. 

Findings of "no significant difference," as reported by Braddock and 
Statler and others (1968) need to be examined further, for such a conclus- 
ion should not be interpreted to mean that no writing growth occurred. 
Diederich has commented that the "notation 'not significant’ does not prove 
that there was not true difference between the two things being compared] 
it only indicates that no true difference was proved" (1964, p, 59), 

Some composition studies have made significant breakthroughs in mea- 
suring writing ability, if not in fact, at least by suggesting useful tech- 
niques. For instance, Dressel, Schmid, and Kincaid (December 1952), while 
failing to find significant results in their study of the effects of writ- 
ing frequency, did, however, employ blind scoring procedures that preserved 
the anonymity of pre- and post-themes. The failure to use a blind scoring 
procedure appears a major defect in the Braddock and Statler experiment al- 
ready summarized (1968, p.13). Diederich (April 1966) has also outlined an 
evaluation procedure that uses scoring blinds. 

According to the Diederich plan, the first step toward the improve- 
ment of essay grading is to discover how widely instructors within a de- 
partment disagree when they all grade the same paper without knowing who 
wrote it or what class level it is supposed to represent. Diederich is 
not overly optimistic about the attainment of high levels of reader re- 
liability; . 



In judging anything as complex as writing ability, 
however, I think it is unrealistic to expect a 
higher average agreement in a department than is 
represented by a correlation of .5 . . , . All 

that is necessary to get it up to a reliability 
of ,8 is four samples of each student’s work, each 
rated independently by two readers, with a third 
rating for papers on which there is substantial 
disagreement (April 1967, p, 582), 

Nevertheless, through a process of evaluation and discussion among raters, 
Diederich suggests that a department could achieve "reasonably uniform 
standards in grading" in about three years ( Ibid , p.584). He also asks 
a provocative question that highlights the need for efforts to achieve 
consensus on grading criteria: "I wonder why we should pretend to be 

able to teach anything like good writing if no two of us can agree even 
this much on what it is" ( Ibid , p. 585) . 
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Kincaid (1953) studied factors affecting variations in the quality 
of student writing and found that neither the content factor, because 
of different assigned topics, nor the pressure introduced by the exam- 
ination situation had any significant effect on the average quality of 
wr iting by student groups of twenty or more. He also found that a single 
paper could provide a valid basis for evaluating writing ability and that 
more reliable information could be obtained from a single pre-test theme 
and a single post-test theme for measuring the overall, or average, group 
improvement . 

Diederich (Decanter 1944) attended to the problem of essay topics. 

He suggests that student writing can be measured provided: 

1. topics assigned are within the student’s compre- 
hension, but not so easy that levels of excellence 
cannot be determined 

2. the form of writing assigned represents the kind of 
writing students may be expected to use in later life 

3. all students write on the same topic, and all papers 
are based on a common set of materials 

4. the essay is written in class 

5. if the mark on the exam is of vital importance to 
the student, the composition is read by two readers 

6. the papers are read without the reader’s knowing 
who wrote them 

7. the papers are marked in accordance with criteria 
formulated and written down in advance 

8. for individual measurement, at least two essays on 
different topics are written to give a reliable mea- 
sure of skill in writing. 

Buxton's study (1958) supports the suggestion made by Diederich 
and others that rater reliability can be increased by experience in 
the development of and practice with a scoring key. Buxton also sup- 
ports the use of class and student blinds as a means of insuring rater 
reliability in essay grading; unfortunately, his results cannot be 
relied on, for his raters apparently were aware of which were pre- 
and which were post-essays. 

The Problem of Scoring Essays 

In all the attempts to measure change in writing ability noted 
here, two problems recur: the problems of reader reliability and 

establishing scoring criteria. Reader variability in the scoring of 
essays' Is well" documented and“may be-regarded -as the- chief obstacle.. 
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to the use of written compositions for measuring change in writing ab- 
ility. Englehart (1969, p. 407) notes that the unreliability of readers 
has been recognized since the 1880s. Since that time, numerous other 
researchers have encountered the same problem (Starch and Elliott, 1912; 
Fostvedt, November 1965; Jewell, Cowley, and Rhum, 1966), yet, despite 
wide recognition of the problem, few successful solutions have been ad- 
vanced. Even one of the most experienced composition researchers, 

Braddock, has stated that most people know that the grading of composi- 
tions is notoriously unreliable. While there are undeniable difficul- 
ties in securing reader reliability, the accuracy of Braddock' s state- 
ment deserves further inquiry* 

The efforts of Diederich, French, and Carlton (1961) well illustrate 
the phenomenon of reader variability. Using three hundred college themes, 
they asked 53 judges in the fields of English, social studies, natural 
sciences, law, writing and editing, and business to judge each theme on a 
nine-point scale. The results show that "ninety-four percent of the themes 
received either seven, eight, or nine of the nine possible grades and no 
paper received less than five different grades from the 53 readers: (Ibid, 
p. 58) . 

While these results show a wide range of reader variability, it is 
significant to note that the ten English instructors in the study had a 
higher mean intercorrelation (.41) with one another than did any other 
of the occupational groups. One might well ask of what use is it to know 
that lawyers and business executives cannot agree on the excellence of 
student compositions. It is significant, however, to note that none of 
the readers was given criteria by which to rate the compositions in t s 
study. Very different results might have occurred if standard criteria 
for scoring had been employed. 

Other researchers have supported the use of essays for measuring 
writing skill, yet their solutions to the problem of reader variability 
have often lacked practicality. Greene and Petty (1963) have suggested 
that reliable essay evaluation requires extensive samples as well as re- 
peated ratings by expert judges to minimize variability. Godshalk , 
Swineford, and Coffman (1966) reached similar conclusions, for they report 
that essay-score reliability is a function of the number of different es- 
says and the number of different readings. They recommend the evaluation 
of five different essay topics, with each topic read by five different 
readers. It is obvious that such an evaluation procedure would take too 
much time; it is therefore not a practical solution to the problem of 
reader variability in the community college. 

The work of Follman and Anderson (1967) offers provocative suggest- 
ions to ameliorate the unreliability of scoring keys. Using upper- 
division college English majors as raters, Follman and Anderson assigned 
ten themes to five groups of raters to determine the intra-reliability 
of five composition scales. Follman and Anderson found that the differ- 
ences among rating groups did not change with the subject matter of the 
essays, and that the essays received substantially the same scores from 
all five rating- groups. Their conclusion is that the high reliability 
across different evaluation procedures "may be due primarily to the homo- 
geneous nature of the raters rather than to a rating system (Ibid , p.19 ) 



This conclusion is particularly notable since one of the evaluation 
procedures, the Everyman's Scale, allowed each rater to use his own 
criteria-yet the second highest reliability score was obtained using 
the Everyman Scale. Follman and Anderson observe that: 

It may now be suggested that the unreliability usually 
obtained in the evaluation of essays occurs primarily 
because raters are to a considerable degree heterogen- 
eous in academic background and have different experi- 
mental backgrounds . . . likely to produce different 

attitudes and values which operate significantly m 
their evaluations of essays ( Ibid , pp. 198-199), 

Their comment on the value of scoring keys is significant: 

The function of a theme evaluation procedure, then, 
becomes that of a sensitizer or organizer of the rater s 
perception and gives direction to his attitudes and 
values; in other words, it points out what he should 
look for and guides his judgment ( Ibid , p.199). 

This suggests the possible value of cooperative grading practices 
by English departments in the community college. Indeed, Diederieh 
has written about the possible values of cooperative grading of 
English compositions: 

It makes the job easier, quicker, and more interesting 
by a division of labor; it puts teachers and students 
on the same side of the fence; it reveals answers to 
many teaching problems; it provides ammunition against 
our critics; and it adds fun and excitement to both 
teaching and learning (1967 # p,579)* 

Nealy (November 1969) even demonstrated that standards for composition 
are "readily communicable" between instructors and their students , 
Surely English instructors within a department ought to be able to 
increase their reliability through practice with standard scoring 
criteria, despite the initial difficulties of reaching consensus. 

The potential merits of composition scales bear repeating in 1970 

These scales are but means toward ends. The most 
important of these ends are: (1) to test impartially 
the various methods of teaching composition by 
measuring their results; (2) to measure these results 
in accurate, objective, stable, and understandable 
terms; (3) to furnish a common basis for comparing 
the same class or school or that of pupils in 
different classes or schools; (4) to classify pupils 
fairly in compostion; (5) to grade them justly 
within their group; (6) to enable teachers to 
discover their reliability in judging the general 
merit of English Composition; and (7) to furnish 
pupils an incentive to self-competition (Hudelson, 

1925) . 
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The literature on composition scales is clearly not definitive. 

The development of practical scoring criteria is needed, for, as 
Diederich comments : 

I honestly believe that almost all experiments con- 
cerning English compost ion that rely on essay grades 
have been conducted with tape measures printed on 
elastic, and that they must be replicated with measures 
in which we can have confidence (1964, p .60) . 

The question remains: What type of scale is superior? Studies 

such as those by Cast (1939 , 1940}, Coward (1952)^ and Nisbet (1955) 
have attempted to determine the superiority of the analytic versus 
the holistic approach to grading essays. While Nisbet found that the 
two methods could produce nearly equivalent results in terms of rater 
reliability, both Cast and Coward found some evidence of the, superior- 
ity of the analytic approach. The scoring method selected should de- 
pend on institutional purpose and philosophy: in the community college, 

with its diverse student population, there is a detailed information 
about program evaluation and improvement. The holistic approach to 
grading essays simply will not yield as much useful information as 
the atomistic approach. 

Other researchers have investigated different point scales. McColly 
and Remstad (October 1965) demonstrated that a four-point scale can be 
used with reliability equal to that of the more time-consuming six-point 
scale, Jewell's (1966, p.19) work supports the utility of the four- point 
scale for scoring essays. 

The use of explicit scoring criteria seems to offer the most promise 
for reducing reader variability in the scoring of written compositions. 
Studies by Torgerson and Green (1953), Diederich, French, and Carlton 
(1961),, Nybert (1966)^ and Hyndman (1969) have explored the identification 
of clusters of grading variables. Their studies should have led to fur- 
ther experiments with scoring keys . The fact that few recent studies 
have employed scoring keys may be attributed to the persistent belief 
in their unreliability. It may also be true that the persistent belief 
in the unreliability of scoring keys stems from a widespread failure to 
experiment with composition scales. 

Summary 

This chapter has reviewed composition research related to the pro- 
blem of measuring changes in writing ability. Both difficulties and 
potential solutions have been noted. The major problem confronting the 
composition researcher is to reduce reader variability. The use of 
blind scoring procedures, of a four-point scale, of a cooperatively 
devised scoring key with clearly defined criteria, of pre- and post- 
themes, the use of practice in grading, and the use of relatively 
inoffensive experimental procedures have been identified as potential 
aids in securing meaningful measures of writing growth. In Chapter 
4, attention will be given to each of these concepts. 



CHAPTER 3 



RATIONALE AND DESIGN 



Rationale 

This design stems from certain definitions and philosophical op- 
inions regarding education. The overriding position is that education 
is a process of moving people from one set of capabilities or tenden- 
cies to another. Within this framework are certain definitions; e.g. , 
learning is changed capability for, or tendency toward, acting in par- 
ticular ways; and instruction is the deliberate sequencing of events 
so that learning occurs. Thus, education is not seen as providing an 
environment in which something of unknown effect may or may not occur; 
rather, it is the bringing about of change. Within the community col- 
lege, it is the instructors who should predict and define the nature of 
that change and assess the effects of their instructional process, yet, 
despite the notation in every college catalog, M The student will learn 
to write effectively , v no one really knows the extent to which writing 
improves, or if, in fact, it improves at all as a result of college at- 
tendance . 

Specifically stated, nine major assumptions underlie this study: 

1, Community college English instructors have an obligation 
to define and assess the effects of their instruction, 

2, Learning is shown by changed capabilities. 

3, It is possible to measure reliably changes in writing 
ability through a comparison of pre- and post- themes written 
a semester apart. 

4, Community college English instructors can devise a valid 
scoring key; with practice, they can use the scoring key 
to grade English compositions reliably, 

5, Such a scoring key should contain several scoring categor- 
ies . 

6, The scoring key should call for analytic distinctions, 
rather than global judgments, 

7, Rater reliability should be measured through an analysis of 
variance, using scores assigned to common compositions, 

8, Scoring blinds (student, college, class, time) assist in 
achieving objective scoring of student themes. 

9, The random assignment of papers from different classes to be 
scored by many scorers helps to distribute the influence of 
any single reader. 

The design of this study is an attempt to know more about the 



effects of composition courses and to solve the major problems as- 
sociatedwith the measurement of writing sklllsFxrstthedeagn 
calls for actual demonstrations of composition skill, and, m this 
resoect has greater face validity than studies that rely on j.n 1 
rect measures of writing skill. The very fact of pre- andpost-tes 
ing allows a defensible comparison of compositions and permits th 
measurement of change in writing ability. 

Second, the design mitigates the causes of reader unreliabil- 
ity in the scoring of essays. Through the use of scoring blinds, 
reader bias regarding authorship, class level, and time of composi- 
tion are controlled. 

Third error introduced by difficult or easy readers is con- 
trolled by random sampling in the distribution of themes for final 
scoring. In this way, scoring error between the pre- and post tests 

is balanced. 



Fourth and finally, variations in scoring are mitigated through 
the definition of and practice with explicit scoring criteria. The 
design reflects the belief that a meaningful assessment of composition 
skill requires the cooperative evaluation of essays according to clear- 
ly defined criteria CDiederich, 1967). Through the use of the coop- 
eratively developed scoring key, reliable measures of writing ability 
are obtained and, since these measures are guided by a common scoring 
key, it is possible to make comparisons between the writing perform- 
nances of students in English composition programs in different col- 
leges . 



Hypotheses 



Two general hypotheses were tested by this research. Each im- 
plies a series of specific hypotheses; 

1. Procedures for scoring English compositions can be developed 
that are reliable, internally consistent, have face validity 
and are meaningful for the evaluation and change of instruc- 
tion. 

2. There are significant differences in the improvement in writ 
ing ability between students enrolled in different colleges. 

Design Procedures 

To seek an assessment of writing ability, student compositions 
were written during the class periods in the first and last weeks of 
the fall semester at three community colleges. Pre- and post-composi- 
tions were collected and matched and given blind code numbers. Blue- 
book covers that identified the student and college were removed. Pre- 
and post-compositions were mixed, and packets of approximately twenty- 
five compositions were then randomly formed and distributed for scoring 



Score sheets were collected by the investigator, and the data were 
then key-punched for processing by computer to determine group means on 
the pre- and post-compositions . A series of Chi-squares were computed 
to measure individual and group scoring changes. Pearson correlation 
co-efficients were computed for the scoring key variables. A test of 
rater reliability was performed on the basis of three common themes that 
were placed in each instructor's packet of themes. 

Selecting the Participants 

The presidents and deans of instruction of several Southern California 
community colleges were asked to participate in this study. Three colleges 
expressed an interest and the design was presented to interested faculty 
members at each college. The only restriction placed on faculty by the in- 
vestigator was that they be assigned to teach a freshman writing course. 

Because one college had considerably more participating instructors 
than the other two, most of the themes came from that institution. 

Selecting the Classes 

Participating instructors were asked to select one or more of their 
classes for use in this experiment, depending on the number of assigned 
writing courses they were scheduled to teach during the fall semester, 
1969-70. Most of the student themes in this study came from the standard 
English course for freshman students . 

Developing the Scoring Key 

Building on the work of previous research, instructors from only one 
college constructed the scoring key, which calls for ratings on each of 
fifteen factors forming three general categories: content, organization, 

Mid mechanics. (See Scoring Key, Figure I) The four-point scale was thought 
to assist instructors to score themes faster than and just as reliably as 
a scale calling for finer distinctions. The scoring ranges from zero for 
a perfect score to three for unacceptable performance for each of the 
fifteen scoring items. 

Checking Rater Reliability 

During the development of the final scoring key, reader reliability 
was checked by having each instructor read and score duplicated compo- 
sitions. Scoring items that failed to produce reader agreement more 
than 75 per cent of the time were modified or rejected. After three re- 
liability trials and before the final scoring, all scoring items met this 
requirement. During the final scoring, each instructor and the investi- 
gator rated three duplicated themes as a final measurement of reader 
reliabi lity . 

Ch posing the Topic 

The two topics selected for the pre- and post-themes were: "What 

makes a good advertisement?" and "What makes a good entertainer?" These 



topics were selected for several reasons. They did not ask for re- 
sponses that might be intensely personal; they did not invite trite 
responses; they did not require the student to understand rhetorical 
terms such as "compare and contrast"; they both called for expository 
writing. 

Instructions to Students 

Clear instructions to students were printed on sheets of paper 
pasted on the covers of the composition books. For each of the two 
composition topics, the instructions were: 

You are to write a composition in this blue book. 

Write in ink on one side of the paper only. 

Write on alternate lines, please. 

Your topic is: What makes a good entertainer ? 

[Repeat the same instructions,] 

Your topic is: What makes a good advertisement? 

Administering the Composition 

During the first week of his course, each instructor randomly dis- 
tributed to his class or classes a number of blue books; approximately 
half of each class then wrote for a class period on each of the topics 
for the pre-test themes. 

During the last week of the course, the instructor distributed to 
his classes blue books on which the investigator had written student 
names from the pre-tests, insuring that each student wrote his post- 
test composition on the topic different from his pre-test. 

Coding, Sampling, and Scoring 

All pre- and post- themes were collected and identifying marks from 
each composition were removed. Names were entered on a master sheet and 
a code number was stamped on the blue book. Code numbers were assigned 
on a random basis to prohibit number sequence from identifying which 
themes were pre- and which were post- test. 

Approximately 500 themes were matched on a pre- and post-basis be- 
cause of student attrition. To reduce the correction load for each 
instructor, 252 pairs of themes were randomly selected for final scoring. 
The results of this study apply only to those students who persevered 
throughout the entire semester of instruction. 

Packets of approximately 25 themes, plus three duplicated themes 
for the reliability check, were then distributed for scoring to each 
participating instructor. Each instructor was asked to score his packet 
of compositions according to the criteria in the cooperatively developed 
scoring key. 

Analysis, findings, and results of this study are discussed in 
Chapter 4, 



CHAPTER 4 



RESULTS 



A, Measurement of Rater Reliability 

While this investigation proposed to measure changes in the writ- 
ing ability of a sample of community college students, it also examined 
the measurement of reader reliability. When error variance or rater 
variance accounts for more of the total variability than does the quality 
of the essays, little can be said about the growth in writing ability of 
students. Reliability in this experiment varies according to the accuracy 
or sameness with which various instructors use the scoring key. 

Three measures of reliability were used for this experiment. One 
was an approximation of a reliability coefficient from the standard 
error of measurement formula; a second was a correlation between an in- 
dependent rater with all other raters on 100 randomly selected essays; 
and the third was an analysis of a comparison of the variance due to the 
rater and error with the variance due to the quality of the essays. 

As part of the procedure, every participating English teacher (i .e. , 
rater) rated three common essays. The variability in their total score 
for these essays as measured by standard deviation was 7.46, 7.79, and 
7,93, The average was 7.8 One can reasonably assume this average stand- 
ard deviation of 7.8 to be a good estimate of the standard error of mea- 
surement for this scale. As a consequence, a reliability coefficient can 
be obtained by working backward using the formula. 

s e = s* / T r r 

where 

S = standard error of measurement 
e 

S = standard deviation of the test 

r = reliability of the test 

Through computation with the standard error of measurement on the 
scores given to the three common themes and the standard deviation for 
total pre- and post-test scores, we can obtain mi estimate of reader 
reliability: 

S o = S/ 1-r 
e 

7.8 » 10,73/ 1-r 
/l-r = .727 
1-r •• .329 
r •= ,471 
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o 



A second estimate of reliability coefficient was obtained by col- 
lecting two readings of a random sample of 100 essays. For this reli- 
ability study , 100 essays were randomly drawn from the 504 essays used 
in the study. These represent all three colleges, both pre- and post- 
test. Each of the raters had read and scored some portion of the 
sample. They were re-read and independently scored by an additional 
rater. The correlations between this independent rater and the first 
reading are given in Table 1. 



Table 1 

Correlation: One Reader With All Others 

N * 100 



Content 


.39 


Organization 


.41 


Mechanics 


.32 


Total 


.36 



Essays randomly drawn from total population of says, t^tb 
pre- and post-test. 



The third procedure for estimating reliability, though unorthodox , 
has most meaning for this type of research* It results in separating 
the overall variance into that portion due To the quality of the essays 
and that portion due to instructor bias plus random error. Using the 
scores assigned by each Instructor to his sample of essays, mean?? and 
standard deviations were computed for each who has read 23 or mor^ 
essays* There were 18 such instructors. This was done for content* 
organization, mechanics, and total. The variance of these instructor 
means was then computed for each sub-area and the total. The results 
appear in Table 2. Remembering that each sample of essays given to 
each Instructor was randomly selected, the variance of these means gives 
a good estimate of the proportion of the total variance due to instruc- 
tor bias plus sampling error * When this variance is subtracted from the 
total variance of the 504 essays, one can determine with fair accuracy 
how much was attributable to the quality of the essays. The proportion 
of variance due to instructor bias and error can be determined by divi- 
ding the variance of the instructor means by the total variance. These 
data are also reported in Table 2. 

B. Measurement of Group Progress 

The results of the comparisons between total sarrt;!© Cleans pre- to 
post-, are shown in Table 3. 
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TABLE 3 







MEANS, 


PRE-TO 


POST- 




N = 252 


PRE- 

Mean 


SD 


POST 

Mean 


SD 


Dif. 


Cont ent 


4.8 


2.4 


4.6 


2.4 


.2+ 


Organization 


11.2 


5.5 


11.3 


5.6 


. 1- 


Mechanics 


6.8 


4.0 


6.2 


3.9 


. 4+ 


Total 


22.8 


10,8 


22.1 


10.7 


.73+ 



Since a score of zero on the rating scale indicates a perfect score, 
a zero mean would indicate a perfect mean score. Thus, declines in mean 
scores from pre-to post-test indicate gains in writing ability, A com- 
parison of total sample means, pre- to post-, indicates that a slight but 
insignificant gain in writing ability was measured (.73+). 

An inspection of the pre-to post- changes in total sample means by 
major subsections of the scoring key shows that slight gains were observed 
for Content (.2+) and for Mechanics (.4+), while Organization shows a 
slight decline (,1-). The magnitude of these mean changes is not signifi- 
cant enough to indicate that any true gains in writing ability were de- 
tected in this experiment. 

A comparison of pre- and post-test means for each of the individual 
colleges reveals no significant differences. Table 4, 5, and 6 report 
these data. Table 4 shows a comparison of pre- and post- means for Col- 
lege A on the major subsections of the scoring key. 



TABLE 4 

COLLEGE A: MEANS, PRE- TO POST- 



N - 191 


PRE- 

Mean 


SD 


POST- 

Mean 


SD 


Dif, 


Content 


4.7 


2,4 


4.6 


2.4 


s 1 + 


Organization 


11.1 


5.5 


11.4 


5.6 


.3- 


Mechanics 


6.5 


4,0 


5, 8 


3.8 


.7+ 


Total 


22.3 


10.8 


21.8 


10.7 


1 5+ 



O 



Table 5 shows a comparison of pre— and post— means for 
College B on the major subsections of the scoring key. 





TABLE 


5 






COLLEGE B: NEAN3, PRE- TO 


POST- 




N = 24 


PRE- 


POST- 






Mean SD 


Mean 


SD 


Dif. 


Content 


5.5 2.0 


5.5 


2,4 


,00 


Organization 


12,2 4,3 


13. 1 


4.8 


.9- 


Mechanics 


9.0 3^_7 


8.9 


4.2 


,1+ 


Total 


26,8 8,0 


27.5 


10.0 


.7- 


Table 6 shows the comparison of pre 
College C on the major subsections of 


- and post- means for 
the scoring key. 




TABLE 6 








COLLEGE C: MEANS, 


PRE- TO POST- 




N * 37 


PRE- 


POST- 






Mean SD 


Mean 


SD 


Dif, 


Content 


4.6 2.4 


4.1 


2.1 


.5* 


Organization 


11.1 6.3 


9.7 


5.8 


1.4+ 


Mechanics 

Total 


7.2 4.1 


6.6 


3.5 


.6+ 


22.9 11,8 


20.4 


10.4 


2.5+ 



While the preceding results show insignificant changes 
in group means, pre- to post-, an analysis of changes item 
by item on the scoring key, pre- to post-, for the total 
sample reveals considerable movement;. Table 7 presents 
these data. An inspection of Table 7 shows that only one- 
third of the students in the sample showed no change from 
their pre- to post- composition scores. The data from this 
analysis of change suggest that the previous comparisons of 
group means, pre- to post-, mask the full extent of the 



changes that took place in student scores. In the Content subsect- 
ion, 4% more students showed gains than declines according to 
changes in totals for this section. In the Organization subsection 
of the scoring key, a comparison of totals shows that an equal num- 
ber of students gained and declined] only 3% of the sample showed 
no change in scores from pre- to post-test in this section. The 
Mechanics subsection shows that only 1% more of the sample declined 
than gained; 11% of the students made no changes in their scores on 
this section from pre- to post- test, 

TABLE 7 

CROSS-TABULATION OF SCORING ITEMS, 

PRE-TO POST-, FOR THE TOTAL SAMPLE 



N = 252 



Scoring Items 


% Gain 


% Decline 


% No Change 


1 Treatment 


38 


33 


29 


2 Knowledge 


37 


31 


32 


3 Diction 


35 


32 


34 


CONTENT TOTALS 


45 


41 


14 


4 Thesis 


39 


36 


25 


5 Plan 


33 


38 


29 


6 Unity 


25 


44 


31 


7 De ve 1 opment 


33 


34 


33 


8 Patterns 


32 


32 


36 


9 Transitions 


32 


32 


36 


10 Logic 


36 


34 


30 


ORGANIZATION 


47 


47 


3 


1 1 Spe 1 ling 


40 


27 


33 


12 Syntax 


37 


28 


35 


13 Punctuation 


36 


35 


29 


14 Major error 


38 


30 


32 


15 Minor error 


36 


31 


33 


MECHANICS TOTALS 


44 


45 


11 


GRAND TOTALS 


49 


48 


3 



O 

ERIC 



S3 



Pearson correlation coefficients were computed for scoring-key 
items to measure the validity of each item in the scoring key. In 
addition, Pearson correlation coefficients were computed for scoring- 
key items to total test scores to provide measures of the rater's 
practice in assigning scores. Table 8 shows the degree to which 
scoring-key items were related to total scores on the pre- . and post- 
test. Only three scoring items show less than a .72 relationship to the 



TA RTP 



CORRELATION COEFFICIENTS FOR SCORING ITEMS. 
TO TOTAL TEST SCORES, PRE- TO POST- 



Scoring 

Item 



Total Score Total Score 

p ye- test Post-test 



1 


Treatment 


. 7838 


2 


Knowledge 


.7691 


3' 


Diction 


. 8101 


4' 


Thesis 


. 7104 


5 


Plan 


.8240 


6 


Unity 


.8150 


7 


Development 


. 7361 


8 


Patterns 


. 8455 


9 


Transitions 


.8290 


10 


Logic 


.7946 


11 


Spelling 


.6098 


12 


Syntax 


.8021 


13 


Punctuation 


.7360 


14 


Major error 


. 7275 


15 


Minor error 


.6916 



.7709 
. 7827 
.8485 
.7359 
.8377 
.849 7 
.7598 
.8335 
.8348 
.8241 
,5345 
.8031 
. 7882 
.6723 
,6417 



total scores: Item 11 Spelling, Item 14 Major Mechanical Errors, 

post-test only, and Item 15 Minor Mechanical Errors, The complete 
scoring key and criteria appear in Figure I. 

Tables 9 and 10 show the intercorrelations between scoring-key 
items, on the pre- and on the post-tests, an analysis useful in id- 
entifying which items may be of little use in the scoring key. 

Chapter 5 presents the investigators’ conclusions and sugges- 
tions for further study. 
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TABLE 9 



INTERCORRELATIONS OF SCORING- KEY ITEMS s PRE-TEST 



2 


3 


4 


5 


6 


1 ,87 


.61 


.57 


.73 


,69 


2 


.61 


.56 


,71 


.69 


3 




.54 


.61 


.57 


4 






,70 


,68 


5 








.85 



6 

7 

8 

9 

10 
11 
12 

13 

14 



7 


8 


9 


10 


11 


12 


,64 


,59 


.61 


.61 


.36 


.50 


,63 


.58 


.57 


.59 


,34 


.49 


.57 


.69 


.65 


.63 


.48 


.73 


.52 


.49 


.56 


.54 


.36 


.48 


.68 


.62 


.68 


.66 


,37 


.54 


.69 


,62 


.69 


.66 


.39 


.54 




.65 


.61 


.55 


.32 


,51 






.76 


.65 


.49 


.70 








.75 


.45 


.63 










.38 


.64 












.53 



15 


14 


15 


45 


.42 


.40 


47 


.38 


,41 


56 


,56 


.53 


41 


.36 


,37 


46 


,46 


.42 


42 


,45 


.41 


43 


,36 


.33 


,66 


.64 


.59 


.53 


.55 


.52 


,53 


.51 


,44 


,47 


.52 


.49 


,65 


.65 


,61 




.70 


.65 






.73 



TABLE 10 

INTERCORRELATIONS OF SCORING-KEY ITEMS, POST-TEST 



2 3 



1 ,81 ,66 

2 ,69 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 



4 


5 


6 


7 


8 


9 


10 


11 


12 


13 


14 


15 


63 


.72 


.72 


,67 


.57 


.61 


.63 


.30 


.50 


.49 


.32 


.31 


62 


.69 


.72 


.68 


.59 


.62 


.66 


.28 


.52 


.48 


.41 


.29 


59 


,72 


.70 


.58 


.67 


,65 


.70 


.40 


.75 


,65 


.53 


.52 




. 70 


.69 


.58 


.56 


.59 


.59 


.27 


.46 


.48 


.37 


,32 






.88 


,67 


,64 


.68 


.70 


,34 


.56 


.54 


.49 


.38 








,69 


.67 


.69 


.71 


,37 


.58 


.56 


,43 


.36 










,62 


.64 


,62 


.30 


.52 


.51 


,37 


.30 












.75 


.65 


,44 


.67 


.68 


.54 


.56 














.76 


.35 


,63 


.62 


.51 


.53 
















.30 


.68 


,62 


.51 


.40 


















.46 


,42 


.37 


.49 




















.74 


,60 


.61 






















,65 


.62 
























.65 





CHAPTER 5 



CONCLUSIONS 



Significant changes in writing ability were not detected in this 
study through a comparison of pre- and post-means for the total sample, 
or for any of the three colleges. A t-test was used to determine the 
significance of the differences between the means for the correlated 
samples in this study. 

An analysis of the individual score changes, pre- to post-, on 
the fifteen scoring items revealed that nearly all student scores 
changed during the semester. 

This experiment supports the use of a scoring key to enable English 
instructors to achieve a high degree of consistency in the grading of 
essays. Most of the observed differences in student compositions are 
apparently true differences and are not attributable to inter-rater un- 
reliabi lty . Further, an analysis of the correlations between the 
scoring items and total test score indicates the homogeneous nature of 
instructor scoring patterns. The relationships between item scores and 
total test scores on the pre-test approximate those ahieved on the post- 
test, with one exception: Item 11 Spelling Errors. 

interpretation of Findings 

It is possible that the four-part scale contributed to the find- 
ing of no change in writing ability in this study. It is also possible 
that the four-way scale allowed for too much discrimination, or too 
wide a range of instructor choice. Another possible explanation for 
the results is that, since most of the students were in the standard 
English course, there were no sharp differences in skill level. 

The most likely explanation is that students do not learn to write 
during eighteen-week courses that concentrate on numerous English- 
language activities. Extensive practice in expository writing, with 
topics similar to those used in this study, might well cause significant 
changes in writing ability if this were the foremost goal of the course. 
The amount and type of writing called for by each of the mixed-bag 
courses in this study vary. It remains to be tested if a semester of 
instruction, concentrated on the problems of expository writing, can 
cause significant gains in writing ability. 

It is also possible that the results can be attributed to the one- 
shot nature of the experiment. It is not known from this study what 
the results would be if English instructors and their students worked 
with a similar scoring key throughout a semester or for longer periods 
of time. Student knowledge of the scoring-key items might well assist 
them to develop their own writing ability. The problem of student 
motivation also remains open to question. While students and instruc- 
tors indicated enthusiasm for the project, they may have regarded the 
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two essays in this project as peripheral to the instructional program in 
their classes. Still the possibility that students do not learn to im- 
prove their writing skills must be admitted* 

In any event, further experiments with this scoring key and these 
procedures are needed to establish a baseline of possible changes that 
might be attributed to highly effective instruction. Even if signifi- 
cant changes had been found through a comparison of group means, it 
would not be possible to call the degree of change satisfactory or 
unsatisfactory without replication. 

A significant finding is that a high degree of rater reliability 
can be achieved through the use of a scoring key. It is possible that 
the cooperative development of the scoring key and the subsequent prac- 
tice in its use contributed to the high degree of reliability attained. 
The reliability of the readers 1 scores was also increased by the scoring 
blinds built into the evaluation procedures* These efforts to limit 
reader bias appear to have been highly successful. 

Rater reliability was also raised by the random sampling procedures 
used to distribute pre- and post-tests* Whatever scoring variation 
existed between the raters on the pre-tests, it can be assumed that it 
also existed during the post-test scoring; in effect, high scorers and 
low scorers cancelled each other out through the random distribution of 
the essays for grading* While this procedure undoubtedly increased 
the objectivity of the results, it also may have contributed to the 
leveling of the gain scores. The utility of broad scoring categories 
is not supported by this investigation, since no significant gains in 
writing ability were determined on the basis of total scores or on 
subsection scores. 

In summary, the high degree of inter-rater reliability estimated 
in this study and the results obtained support the conclusion that 
the students did not significantly improve their writing abilities 
during the eighteen-week period of instruction. While a variety of 
factors may have contributed to these results, the evidence calls into 
question studies in which multiple blind-scoring techniques were lack- 
ing and in which substantial gains in ability were recorded. Rater 
bias simply cannot be controlled when readers know whether the composi- 
tions were written before or after instruction. 



Strenghts and Weaknesses of the Study 

A major strength of this study is that it demonstrates that 



reader reliability can be improved through the use of a cooperatively 
developed scoring key. The study also supports the us© of these eval- 
uation procedures as an in-service training program. Win le their 
benefits in in-service training are not directly observable m this stu y, 
the fact that twenty-one English instructors from three colleges could 
attain high reliability suggests the value of cooperative grading of 
essays 4 

Another strength of this design is the relative ease of carrying 
out the procedures once a scoring key is developed. The amount of time 
required to collect, identify, and redistribute the compositions for 
scoring is minimal and need not cut into instructor time for these duties 
can be delegated to clerical help. The data analysis can be programmed 
easily by the director for research and his staff. It is possible, ex 
eluding the development of the scoring key, that the actual scoring of 
themes would take less than the normal time, since the instructor would 
know precisely what he is looking for in the compositions. In any event, 
the use of this evaluation technique as an in-service training device 
and as a way to evaluate instruction is more defensible than current 
grading practices, which yield few benefits to anyone. 

This study has two important weaknesses. Because instructors 
handled a variety of writing problems during the semester, it cannot 
be stated that instruction concentrated on the type of writing evaluated 
in this study. Because the study was coordinated by outsiders, this 
weakness could not be lessened. Even under the best of conditions, it 
might be difficult to find an English Department that would agree to em 
phasize expository writing almost exclusively. If this had been done, 
however, the results of the study might be quite different. 

The second weakness is that the problem of student motivation was 
not given enough attention. While the test topics were relatively easy, 
no attempt was made to motivate the students to write their best for 
the study. Only in College C did the instructors treat the writing 
as a graded part of their regular elasswoxk. If the post-tests had 
been the culminating exercise in the course at each of the colleges, 
the results of this study might again have been quite different . 
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Figure I: The Scoring Key 



DEGREE TO WHICH CRITERIA ARE MET 





Meets all 


Average 


Minimum 

acceptable 


Unaccep table 




criteria 


performance 


performance 


performance 




(0 Point) 


(1 Point) 


(2 Points) j 


(3 Points) 



CORING DIRECTIONS 



EACH PAPER SHOULD BE CHECKED (*0 FOR FACH ITEM 1-15 



ONTENT 



Treatment of 
subj ect 

Knowledge of 
subject 

Diction 



Unusually creative 
and perceptive 

Obviously informed by ex- 
perience or study 

Appropriate to content and 
style; exact word choice 



RGANIZATION 



Thesis state- 
ment 



Focus and purpose clearly 
identified 



, Design or plan 
* Overall unity 



Coherent and/or rhetorical 
ob j ectives achieved 



All paragraphs contribute 
to development of thesis 



Development of 
individual para- 



Sufficiently developed 
with relevant, specific 
detail 



Sentence pat- 
terns 



Coherent and varied pat- 
terns; good subordination/ 
coordination 



. Transitions 
, Logic 



All contribute to logical 
progress of ideas 

All ideas expressed ratio- 
nally and without fallacy 



ECHANICS 



Spelling errors 



None 



Syntax and word 
choice 



Lucid, orderly arrangement; 
diction appropriate to con- 
text 



Sentence struc- 
ture and punc- 
tuation and gross 
error penalties 

Major mechanical 
errors 



Clear and grammatically 
correct 



None 



Minor mechanical 

O i 

.ERIC 



None 



GROSS ERROR PENALTIES: Sentence Fragment - +8 
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